Understanding and Addressing Bias in Grading: Progress at Washington University School of Medicine.

In 2020, the authors published work examining disparities in clerkship grading and found students who identify as White were more likely to receive honors grades compared with students from races/ethnicities traditionally underrepresented in medicine. Through a quality improvement approach, the authors identified the following 6 areas where improvements could mitigate grading disparities and, therefore, put processes in place to: ensure equitable access to examination preparation resources, change student assessment, develop medical student curriculum interventions, improve the learning environment, change house staff and faculty recruitment and retention practices, and provide ongoing program evaluation and continuous quality improvement processes to monitor for success. While the authors cannot yet be sure that they have achieved their goal of promoting equity in grading, they believe this evidence-based, multipronged intervention is a clear step in the right direction and encourage other schools to consider a similar approach to tackling this critically important problem at their own institutions.

Cognitive Load Assessment Scales in Simulation: Validity Evidence for a Novel Measure of Cognitive Load Types.

INTRODUCTION: Cognitive load (CL) theory provides a framework to inform simulation instructional design. Reliable measures of CL types (intrinsic [IL], extraneous [EL], and germane load [GL]) in simulation are lacking. We developed the novel Cognitive Load Assessment Scales in Simulation (CLAS-Sim) and report validity evidence using Kane's framework. METHODS: This quasi-experimental study tested the effect of a segmented/pause-and-debrief or standard/end-of-case-debrief intervention on pediatric residents' performance and self-rated CL in 2 complex- and simple-case simulations. After each simulation, participants completed 22 items measuring CL types. Three validity inferences were examined: scoring (instrument development and principal component analysis); generalization (internal consistency reliability of CL-component items across cases); and extrapolation [CLAS-Sim correlations with the single-item Paas scale, which measures overall CL; differences in primary task performance (high vs low); and discriminant validity of IL under different instructional-design conditions]. RESULTS: Seventy-four residents completed both simulations and postcase CLAS-Sim measures. The principal component analysis yielded 3 components: 4-item IL, 4-item EL, and 3-item GL scales (Cronbach's α, 0.68-0.77). The Paas scores correlated with CLAS-Sim IL and total CL scores in both cases ( rs range, 0.39-0.70; P ≤ 0.001). High complex-case performers reported lower IL and total CL (analyses of variance, each P < 0.001). In multivariate analyses of variance, CLAS-Sim IL, GL, and total CL varied across both cases by arm (each P ≤ 0.018); the segmented-debrief arm reported lower IL than the standard-debrief arm in both cases (each P ≤ 0.01). CONCLUSIONS: The CLAS-Sim demonstrates preliminary validity evidence for distinguishing 3 CL types but requires further study to evaluate the impact of simulation-design elements on CL and learning.

B.2.16 is a non-lethal modifier of the Dark82 mosaic eye phenotype in Drosophila melanogaster.

Genetic screens have been used to identify genes involved in the regulation of different biological processes. We identified growth mutants in a Flp/FRT screen using the Drosophila melanogaster eye to identify conditional regulators of cell growth and cell division. One mutant identified from this screen, B.2.16, was mapped and characterized by researchers in undergraduate genetics labs as part of the Fly-CURE. We find that B.2.16 is a non-lethal genetic modifier of the Dark82 mosaic eye phenotype.

Workplace-Based Assessments Using Pediatric Critical Care Entrustable Professional Activities.

BACKGROUND: Workplace-based assessment (WBA) is critical to graduating competent physicians. Developing assessment tools that combine the needs of faculty, trainees, and governing bodies is challenging but imperative. Entrustable professional activities (EPAs) are emerging as a clinically oriented framework for trainee assessment. OBJECTIVE: We sought to develop an EPA-based WBA tool for pediatric critical care medicine (PCCM) fellows. The goals of the tool were to promote learning through benchmarking and tracking entrustment. METHODS: A single PCCM EPA was iteratively subdivided into observable practice activities (OPAs) based on national and local data. Using a mixed-methods approach following van der Vleuten's conceptual model for assessment tool utility and Messick's unified validity framework, we sought validity evidence for acceptability, content, internal structure, relation to other variables, response process, and consequences. RESULTS: Evidence was gathered after 1 year of use. Items for assessment were based on correlation between the number of times each item was assessed and the frequency professional activity occurred. Phi-coefficient reliability was 0.65. Narrative comments demonstrated all factors influencing trust, identified by current literature, were cited when determining level of entrustment granted. Mean entrustment levels increased significantly between fellow training years (P = .001). Compliance for once- and twice-weekly tool completion was 50% and 100%, respectively. Average time spent completing the assessment was less than 5 minutes. CONCLUSIONS: Using an EPA-OPA framework, we demonstrated utility and validity evidence supporting the tool's outcomes. In addition, narrative comments about entrustment decisions provide important insights for the training program to improve individual fellow advancement toward autonomy.

A Novel Approach to Assessing Professionalism in Preclinical Medical Students Using Multisource Feedback Through Paired Self- and Peer Evaluations.

Phenomenon: Professionalism is integral to the role of the physician. Most professionalism assessments in medical training are delayed until clinical rotations where multisource feedback is available. This leaves a gap in student assessment portfolios and potentially delays professional development. APPROACH: A total of 246 second-year medical students (2013-2015) completed self- and peer assessments of professional behaviors in 2 courses following a series of Team-Based Learning exercises. Correlation and regression analyses were used to examine the alignment or misalignment in the relationship between the 2 types of assessments. Four subgroups were formed based on observed patterns of initial self- and peer assessment alignment or misalignment, and subgroup membership stability over time was assessed. A missing data analysis examined differences between average peer assessment scores as a function of selective nonparticipation. FINDINGS: Spearman correlation demonstrated moderate to strong correlation between self-assessments completed alone (no simultaneous peer assessment) and self-assessments completed at the time of peer assessments (ρ = .59, p < .0001) but weak correlation between the two self-assessments and peer assessments (alone: ρ = .13, p < .013; at time of peer: ρ = .21, p < .0001). Generalized estimating equation models revealed that self-assessments done alone (p < .0001) were a significant predictor of self-assessments done at the time of peer. Course was also a significant predictor (p = .01) of self-assessment scores done at the time of peer. Peer assessment score was not a significant predictor. Bhapkar's test revealed subgroup membership based on the relationship between self- and peer ratings was relatively stable across Time 1 and Time 2 assessments (χ2 = 0.83, p = .84) for all but one subgroup; members of the subgroup with initially high self-assessment and low peer assessment were significantly more likely to move to a new classification at the second measurement. A missing data analysis revealed that students who completed all self-assessments had significantly higher average peer assessment ratings compared to students who completed one or no self-assessments with a difference of -0.32, 95% confidence interval [-0.48, -0.15]. Insights: Multiple measurements of simultaneous self- and peer assessment identified a subgroup of students who consistently rated themselves higher on professionalism attributes relative to the low ratings given by their peers. This subgroup of preclinical students, along with those who elected to not complete self-assessments, may be at risk for professionalism concerns. Use of this multisource feedback tool to measure perceptual stability of professionalism behaviors is a new approach that may assist with early identification of at-risk students during preclinical years.

Entrustable Professional Activities: A Faculty Development Workshop for Adding EPAs Into Your Medical Trainee's Assessment Portfolio.

INTRODUCTION: The last 3 decades have seen significant changes in medical education and corresponding assessment of medical trainees. Competency-based medical education provided a more comprehensive model than the previous time-based process but remained insufficient. Introduced in 2005, entrustable professional activities (EPAs) offer a more robust curriculum development and assessment process, especially in regard to clinician-oriented workplace-based assessments. Despite their intuitive match with decisions made in the clinical environment daily by clinicians, the development of specialty-specific EPAs and corresponding culturally situated assessment tools has lagged. METHODS: To address this gap, a 90-minute faculty development workshop was created to introduce faculty to EPAs and their assessment and to provide hands-on practice developing and using EPAs. RESULTS: Previous facilitations of this workshop received favorable responses from participants regarding level of detail, understanding of the content, and intent to employ EPAs at their own institutions. DISCUSSION: Implementation of EPAs into the assessment portfolio of medical trainees following this workshop will maximize confidence in determining when a trainee is ready for independent practice.

Using reflection to influence practice: student perceptions of daily reflection in clinical education.

PURPOSE: Reflection is a key element in learning from experience, but the impact of most programmes of reflection on daily practice remains unclear. We investigated students' perceptions of adding a daily written reflection assignment to a clinical rotation. METHODS: Third-year medical students on a single two-week rotation completed daily reflections analyzing their performance. Programme evaluation used a 33-question anonymized survey. Quantitative data were summarized and qualitative responses coded for recurring themes. RESULTS: Twenty-six students completed the survey (90 % response rate). Eighty-five percent of students felt that the daily reflections had a positive impact on their learning from clinical experience. Seventy-seven percent of students reported that the programme changed their awareness of their thoughts and actions, and 80 % felt that it improved their recall of experiences. A greater sense of mindfulness and focus on self-improvement were major themes that emerge from students' descriptions of the role of daily reflections in their learning. CONCLUSION: Overall, daily reflections demonstrated a positive learning influence. This exploratory study suggests students may benefit from more frequent, short reflections as opposed to more typically spaced reflective assignments.

Developing a Tool to Assess Placement of Central Venous Catheters in Pediatrics Patients.

BACKGROUND: Pediatric critical care medicine requires the acquisition of procedural skills, but to date no criteria exist for assessing trainee competence in central venous catheter (CVC) insertion. OBJECTIVE: The goal of this study was to create and demonstrate validity evidence for a direct observation tool for assessing CVC insertion. METHODS: Ten experts used the modified Delphi technique to create a 15-item direct observation tool to assess 5 scripted and filmed simulated scenarios of CVC placement. The scenarios were hosted on a dedicated website from March to May 2013, and respondents recruited by e-mail completed the observation tool in real time while watching the scenarios. The goal was to obtain 50 respondents and a total of 250 scenario ratings. RESULTS: A total of 49 pediatrics intensive care faculty physicians (6.3% of 780 potential subjects) responded and generated 188 scenario observations. Of these, 150 (79.8%) were recorded from participants who scored 4 or more on the 5 scenarios. The tool correctly identified the expected reference standard in 96.8% of assessments with an interrater agreement kappa (standard error) = 0.94 (0.07) and receiver operating characteristic = 0.97 (95% CI 0.94-0.99). CONCLUSIONS: This direct observation assessment tool for central venous catheterization demonstrates excellent performance in identifying the reference standard with a high degree of interrater reliability. These assessments support a validity construct for a pediatric critical care medicine faculty member to assess a provider placing a CVC in a pediatrics patient.

Effects of Team-Based Learning on short-term and long-term retention of factual knowledge.

PURPOSE: We investigated the effect of Team-Based Learning (TBL) on long-term retention of knowledge in comparison to a traditional curriculum. METHODS: As TBL was incorporated into our curriculum in the 2008-2009 academic year, students were compared with those who received the traditional curriculum the year prior. Students in both the groups completed multiple-choice knowledge test at four time points spanning two years. Test performance was compared at each time point to assess changes in knowledge retention as a function of time. RESULTS: Baseline knowledge did not differ significantly between the TBL and control groups [51% versus 46%; t(84) = 0.91, p = 0.37, d = 0.20]. Performance improved after the course for both the groups, but was significantly higher in the TBL group [79% versus 59%; t(84) = 4.96, p = 0.000004, d = 0.95]. However, when assessed prior to the pediatrics clerkship, learning gains from TBL had largely disappeared and the small difference in performance was not significant [57% versus 51%; t(84) = 1.51, p = 0.14, d = 0.32]. CONCLUSION: Incorporating TBL into the pre-clinical pediatrics curriculum led to large gains in knowledge over the short-term, but these gains did not persist. Further research should focus on extending the impact of TBL on long-term knowledge retention.

Developing Professionalism via Multisource Feedback in Team-Based Learning.

UNLABELLED: CGEA 2015 CONFERENCE ABSTRACT (EDITED). A Novel Approach to Assessing Professionalism in Preclinical Medical Students Using Paired Self- and Peer Evaluations. Amanda R. Emke, Steven Cheng, and Carolyn Dufault. CONSTRUCT: This study sought to assess the professionalism of 2nd-year medical students in the context of team-based learning. BACKGROUND: Professionalism is an important attribute for physicians and a core competency throughout medical education. Preclinical training often focuses on individual knowledge acquisition with students working only indirectly with faculty assessors. As such, the assessment of professionalism in preclinical training continues to present challenges. We propose a novel approach to preclinical assessment of medical student professionalism to address these challenges. APPROACH: Second-year medical students completed self- and peer assessments of professionalism in two courses (Pediatrics and Renal/Genitourinary Diseases) following a series of team-based learning exercises. Assessments were composed of nearly identical 9-point rating scales. Correlational analysis and linear regression were used to examine the associations between self- and peer assessments and the effects of predictor variables. Four subgroups were formed based on deviation from the median ratings, and logistic regression was used to assess stability of subgroup membership over time. A missing data analysis was conducted to examine differences between average peer-assessment scores as a function of selective nonparticipation. RESULTS: There was a significant positive correlation (r = .62, p < .0001) between self-assessments completed alone and those completed at the time of peer assessment. There was also a significant positive correlation between average peer-assessment and self-assessment alone (r = .19, p < .0002) and self-assessment at the time of peer assessment (r = .27, p < .0001). Logistic regression revealed that subgroup membership was stable across measurement at two time points (T1 and T2) for all groups, except for members of the high self-assessment/low peer assessment at T1, who were significantly more likely to move to a new group at T2, χ(2)(3, N = 129) = 7.80, p < .05. Linear regression revealed that self-assessment alone and course were significant predictors of self-assessment at the time of peer assessment (Fself_alone = 144.74, p < .01 and Fcourse = 4.70, p < .05), whereas average peer rating, stage (T1, T2) and academic year (13-14, 14-15) were not. Linear regression also revealed that students who completed both self-assessments had significantly higher average peer assessment ratings (average peer rating in students with both self-assessments = 8.42, no self-assessments = 8.10, self_at_peer = 8.37, self_alone = 8.28) compared to students who completed one or no self-assessments (F = 5.34, p < .01). CONCLUSIONS: When used as a professionalism assessment within team-based learning, stand-alone and simultaneous peer and self-assessments are highly correlated within individuals across different courses. However, although self-assessment alone is a significant predictor of self-assessment made at the time of assessing one's peers, average peer assessment does not predict self-assessment. To explore this lack of predictive power, we classified students into four subgroups based on relative deviation from median peer and self-assessment scores. Group membership was found to be stable for all groups except for those initially sorted into the high self-assessment/low peer assessment subgroup. Members of this subgroup tended to move into the low self-assessment/low peer assessment group at T2, suggesting they became more accurate at self-assessing over time. A small group of individuals remained in the group that consistently rated themselves highly while their peers rated them poorly. Future studies will track these students to see if similar deviations from accurate professional self-assessment persist into the clinical years. In addition, given that students who fail to perform self-assessments had significantly lower peer assessment scores than their counterparts who completed self-assessments in this study, these students may also be at risk for similar professionalism concerns in the clinical years; follow-up studies will examine this possibility.